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ABSTRACT 


Online classes, hybrid or blended classes, and web-facilitated classes all employ online 
materials, which instructors and students are using at ever-increasing rates. Accordingly, 
instructors should continuously evaluate the effectiveness of such online materials. This paper 
sets forth the author’s initial attempts to analyze a business course pursuant to items in the 
Quality Matters (QM)™ Rubric designed by an organization whose mission is to certify the 
quality of online and blended courses. The paper illustrates how the author was able to begin to 
improve the business course with the assistance of the Rubric. The paper also illustrates how 
student input can significantly assist the QM review process. 
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INTRODUCTION 
Online Learning 

For the last ten years, the Sloane Consortium has tracked online learning in U.S. higher 
education. (Allen & Seaman, 2013) Sloane defines online courses as those in which 80 to 100 
percent of the course content is delivered online, usually with no face-to-face meetings of 
instructors and students. It defines hybrid or blended courses as those in which 30 to 79 percent 
of the course content is delivered online, typically with online discussions and a reduced number 
of face-to-face meetings. Face-to-face courses are those in which 0 to 29 percent of the content 
is delivered online with a full number of face-to-face meetings. Face-to-face courses are 
subdivided into traditional courses (which use no online technology) or web-facilitated courses 
(which employ various types of online materials). (Allen & Seaman, 2013, p. 7) 

Sloane’s latest survey found that by Fall 2012, an all-time high of 32% of students had 
taken at least one online course in their college career. The latest survey also found that the 
number of students taking at least one online course increased by over 570,000 in the last year to 
a new total of 6.7 million. Moreover, the online enrollment as a percent of total enrollment at 
degree-granting postsecondary institutions has increased from 9.6% to 32% in 10 years. (Allen & 
Seaman, 2013, pp. 4-5) Undergraduate business students enrolled in online courses and online 
programs at an even higher rate. (Weyant, 2013, p. 2). 

The Sloane survey documents the dramatic increase in online courses. Similarly, 
increases have occurred in hybrid/blended and web-facilitated courses, which also use online 
materials. One example pertains to the growing trend toward flipped courses, in which 
instructors deliver certain course materials outside of class time via online materials. The 
instructor then is able to use class time to engage more active learning, including problem 
solving, collaboration with peers, and guidance from the instructor. (Bergmann & Sams, 2012) 
The growth of flipped learning is illustrated by the Flipped Fearning Network website for 
educators, which started out with 2,500 members in January, 2012 and which grew to 12,000 
educators by March, 2013. (Bergmann & Sams, 2013 White Paper, p. 3) 

Advantages of using online materials for flipped learning are similar to using them for 
other web-facilitated courses, hybrid/blended courses, and online courses. Online materials 
provide flexibility, elimination of geographical barriers, improved convenience, and 
effectiveness for individualized and collaborative learning. (Pinto & Anderson, 2013, p. 2) 

They also assist with priming (motivating students by getting them familiar with a topic), 
pretraining (spreading out the cognitive load on students), and enabling diverse learners (by 
means of self-pacing). (Bergmann & Sams, 2013 Review, pp. 7-9) 

Online materials, however, are not necessarily helpful just because they are materials 
online. Students, for example, are neutral with respect to their expectations regarding online 
materials. (McCarty, Bennett, & Carter, 2013, p. 5) Sometimes the materials work and 
sometimes they do not. (Pinto & Anderson, pp. 6-7) Certain results from the Sloane survey 
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indicate that online materials and online courses are improving. In the 2003 survey, 57.2 percent 
of academic leaders rated the learning outcomes in online education as the same or superior to 
those in face-to-face education, and that number is now 77.0 percent. (Allen & Seaman, 2013, p. 
5) However, the survey also shows that only 30.2 percent of chief academic officers believe 
their faculty accept the value and legitimacy of online education, which is lower than the rate in 
2004. (p. 6) Faculty do have concerns about the assessment of online instruction. In response to 
the question as to whether “my institution has good tools in place to assess the quality of online 
instruction,” 37.5% of faculty who teach on line strongly agreed or agreed, compared with only 
19.9% of the faculty who do not teach online. Conversely, 49% of faculty who teach on line and 
50% of faculty who do not teach on line strongly agreed or agreed that the institution has good 
tools in place to assess the quality of face-to-face instruction, (p. 39) 

Quality Matters 

The Quality Matters (QM) not-for-profit organization (developed by MarylandOnline 
with funding from FIPSE - the Fund for Improvement of Post-Secondary Education) recognizes 
the need to evaluate and improve online instruction and offers proprietary tools for quality 
assurance for online instruction via training, peer review, and certification. QM current has more 
than 600 institutional subscribers and 22,000 faculty and staff members in 46 states. QM 
provides guidance for quality control of online instruction via three main mechanisms. 
(MarylandOnline, 2013). 

First, it provides the QM Rubric Workbook for Higher Education, specifically formulated 
to evaluate the online course design of online and hybrid/blended courses, but also very useful to 
instructors who teach web-facilitated courses. The QM Rubric is based (and periodically 
updated) on research-supported and public best practices. The Rubric contains general and 
specific review standards with a scoring system along with annotations that contain explanations 
and examples. (MarylandOnline, 2013). 

Second, QM provides for a peer review process whereby trained QM peer reviewers 
provide constructive and specific comments for course strengths and areas of improvement. The 
process for reviewing a course, with the ultimate goal being QM certification, involves three 
reviewers with online teaching experience and training. The Master Reviewer serves as Team 
Chair and is joined by at least one subject matter expert and at least one reviewer external to the 
institution offering the course under review. (MarylandOnline, 2013). 

Third, QM provides various professional development opportunities on-site, on-line, and 
via Blackboard Collaborator to prepare faculty to design and improve courses and to hold QM 
roles. For example, the author first learned of QM via an institutional faculty development 
session to assist faculty with their online courses and then attended a workshop on applying the 
QM Rubric. The author is currently self-assessing online course materials in the hopes of 
eventually getting courses QM-certified. (MarylandOnline, 2013). 
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QM AND LITERATURE REVIEW 

QM has conducted its own literature review of the assessment of online instruction. Such 
review, however, is by no means exhaustive. Many excellent books of teaching instruction apply 
just as well to online materials as they do to face-to-face instruction. For example, in preparing 
online materials, the author has made great use of Teaching with Style by Anthony Grasha 
(1996) as well as the more recent Teach Like a Champion by Doug Lemov (2010). Books 
specifically written with online materials in mind can also be very helpful: Designing Courses 
and Teaching on the Web: A “How-To” Guide to Proven, Innovative Strategies by Mercedes 
Maria Fisher (2003). 

An even more specific book is E-leaming in the 21st Century by D. Randy Garrison 
(2011, 2nd ed.), which sets forth the well-developed community of inquiry framework involving 
social presence (communication among mutually supportive group members), cognitive presence 
(learning via reflection and discourse), and teaching presence (intentional guidance by the 
instructor). One author recently suggested that the community of inquiry model serves well as a 
pedagogical model for online instructors and analyzed his own undergraduate human resource 
management course with the model. (Weyant, p. 2) 

The Sloan Consortium has developed a much more specific and comprehensive 
assessment for online instruction than the community of inquiry model. Based on a study by 43 
experts in online education administration, Sloane features A Quality Scorecard for the 
Administration of Online Education Programs. The Scorecard has 70 equally-weighted items in 
the nine areas of institutional support, technology support, course development and instructional 
design, course structure, teaching and learning, social and student engagement, faculty support, 
student support, and evaluation and assessment. (Lorenzetti, 2013) 

Even though the Scorecard was designed to assess online instruction for institutions, 
rather than individual courses, many of its categories and items apply well to individual courses. 
For example, under the category of institutional support, policies and guidelines should be in 
place for all the courses at an institution to authenticate that students enrolled in the online course 
are the students completing the course work. Many of the items under the categories of course 
development and instructional design (e.g. measurable learning objectives), course structure (e.g. 
syllabus), teaching and learning (e.g. student-to-student interaction and faculty-to-student 
interaction) are also items under the QM Rubric. (Lorenzetti, 2013) 

The QM Rubric, as with the community of inquiry framework and the Sloan scorecard, is 
well-supported by research. An advantage of the QM Rubric is that it is more comprehensive 
and specific than the community of inquiry framework (Hall, 2010) and more specifically 
applicable to individual courses than the Sloan scorecard. Another advantage of the QM Rubric 
is that it is routinely updated with editions for 2005, 2006-07, 2008-2010, and 2011-13. 

Updating occurs pursuant to updated literature reviews. The literature review for the 2008-2010 
edition contained over 100 articles (MarylandOnline, 2008) and the literature review for the 
2011-13 edition contained nearly 150 articles. (MarylandOnline, 2011). 
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The current QM Rubric contains eight key areas (General Standards) of review and 41 
Specific Review Standards. Twenty-one of the Specific Review Standards are essential 
standards; absence of any of them will cause a course to not be certified until the Standard is 
met. The 41 Standards are accompanied by detailed annotations and examples of good practice. 
The eight General Standards pertain to the following areas: 

1. Course Overview and Introduction 

2. Learning Objectives (Competencies) 

3. Assessment and Measurement 

4. Instructional Materials 

5. Learner Interaction & Engagement 

6. Course Technology 

7. Learner Support 

8. Accessibility 

The QM Rubric focuses on the concept of “alignment” for General Standards 2-6, which 
must work together to facilitate student achievement of desired learning outcomes. 
(MarylandOnline, 2013) Each of the General Standards standards is supported by articles set 
forth in the previously-mentioned literature reviews, and this paper does not propose to review 
that support in detail. However, certain articles pertaining to the key concept of alignment bear 
special mention. Swan, Matthews, Bogle, Boles, and Day (2010) found that their QM revision to 
map objectives to outcomes resulted in better student outcomes. Similarly, Kirkwood and Price 
(2008) found that assessment items should align with learning outcomes to produce better 
learning. This author, for example, is currently studying whether student performance on 
multiple choice reviews and quizzes correlates with student performance on application-type 
questions. 

Instructors also need to align their online instructional materials with their learning 
objectives. Li and Liu (2005), with their Online Top-Down Modeling Model, found that 
instructors should consider the objective/ends/task before the materials/means/tool. Similarly, 
Picciano (2009) proposed a Blending with Purpose model wherein instructors first carefully 
consider their objectives and then determine how to apply the technologies that will work best 
for them. 

Once instructors have applied QM and successfully aligned their learning objectives, 
assessment tools, instructional materials, learner interaction, and course technology, they should 
clarify such alignment to their students. Preston et al. (2010) found that faculty should explain 
what role lectures, technology, and other activities play in the learning process, i.e., how 
everything lines up for each unit. 

APPLICATION OF THE QM RUBRIC 

Instructors who want to apply the most completely researched and recently revised QM 
Rubric consisting of the eight General Standards and 41 Specific Review Standards can access it 
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via institutional or individual membership in QM. This paper sets forth the author’s application 
of the first two Specific Review Standards within General Standard 1 to a legal environment of 
business course at a Midwestern regional comprehensive university with primarily traditional, 
resident students. Most all of these students have had significant access to online material in 
web-facilitated courses, but many have not taken a hybrid/blended or completely online course. 

In applying the two Specific Review Standards, the author surveyed students (100 responses 
from mostly business students) for their opinions as to which specific actions to implement. 

General Standard 1 is entitled Course Overview and Introduction and states: “The overall 
design of the course is made clear to the student at the beginning of the course.” Specific 
Review Standard 1.1 states: “Instructions make clear how to get started and where to find various 
course components.” One of the advantages of an instructor applying the QM Rubric to courses 
is that the instructor is required to look at the course from a fresh perspective. For example, in 
reviewing Standard 1.1 on how to get started, the author pulled up the course home page in 
Moodle and noticed that in the prior year’s conversion from Blackboard, there were several 
navigation panes that were unnecessary and a few that were empty. In previously reviewing the 
home page, the author without specific QM guidance had simply noted that the usual materials 
were present, but did not notice the extra panes. Students new to the course, however, would 
wonder why the panes were there, which would likely impede their efficient navigation of the 
materials. 

The annotations and examples that come with Standard 1.1 suggest various ways to help 
students get started in a course, so the author asked students in an anonymous survey to rank 
some of the options in order of preference. The students also had the opportunity to comment 
upon their selections. The survey ranking item was set forth as follows: 

Please rank the following from the most preferred way (1) to the least preferred 

way (6) to get students started in a course and to help them find various course 

components. 

1) Provide a link at the top of the course home page to the course syllabus that 
would contain the course components. 

2) Provide a “Read Me First” or “Start Here” button that directs students to the 
course syllabus that contains the course components. 

3) Provide a statement that provides directions on how to navigate the course 
components. 

4) Provide a diagram setting forth the course components. 

5) Provide a video setting forth a course tour of the course components. 

6) Provide a scavenger hunt assignment that requires students to investigate the 
course components. 

The options to get students started generally range from the simplest option to the most 
elaborate option and that turned out to be the order of preference selected by the students. 
Typical student comments stated a preference for “quick and simple” and “things that do not take 
up extra time/extra space” and an aversion to options that “would be cumbersome” or “require 
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too much time.” Some comments, however, favored the visually-based options 4-6. For 
example, one student said, “students will more likely want to watch a video or do an activity than 
read directions so that they can see exactly what to do.” 

Based on the survey rankings and student comments, the author plans to add a start 
button pursuant to option 2. Although option 1 was most popular (with 88% of students 
selecting it as one of their top 3 preferences), option 2 was almost as popular (with 81% of 
students selecting it as one of their top 3 preferences). The author will also consider adding 
option 4, 5, or 6, as something that more visual learners may use if they wish; an average of one 
fourth of those students surveyed listed options 4, 5, and 6 as among their top 3 options. The 
author will not require students to use the visual options, though, because of comments by most 
students that they prefer the simple, time-saving methods rather than the more complex methods 
that would be “a hassle” or even “extremely annoying.” 

The author also solicited student input with respect to Specific Review Standard 1.2., 
which states “Students are introduced to the purpose and structure of the course.” Suggestions 
from the annotations to this standard include those pertaining to the format for the online 
syllabus and/or other online materials detailing the purpose and structure of the course. 
Accordingly, the author asked the students for their preferences regarding a course syllabus as 
follows. 

Please rank the following from the most preferred way (1) to the least preferred 

way (4) to inform students about a course and its course components: 

1) Provide students a link to the syllabus that will come up in downloadable 
document format. 

2) Provide students a link to the syllabus that will come up in online format. 

3) Provide students a link to the syllabus that will come up in online format 
that has a table of contents hyperlinked to sections within the syllabus. 

4) Provide students hyperlinks to various online sections, which make the 
traditional syllabus structure unnecessary. 

Students’ first choice was option 1 because they wanted to print the syllabus and refer to 
it at times they were not connected to the Internet. Options 2 and 3 essentially tied for second 
choice, and Option 4 lagged far behind. Based on the survey rankings and student comments, 
the author plans to provide for option 1 and option 3. Although option 1 was students’ favorite 
option (with 70% of students selecting option 1 as one of their top two choices), options 2 and 3 
were also popular (with 59% selecting option 2 as one of their top two choices and 58% selecting 
option 3 as one of their top two choices). Having a syllabus that has a table of contents 
hyperlinked to various sections makes sense considering the number of pages in many syllabi. 
The author will not completely forgo the traditional syllabus structure pursuant to option 4 
because the surveyed students are not yet ready for that (with only 12% of students selecting 
option 4 as one of their top two choices). 

The QM Rubric does not suggest that instructors survey students when applying the 
Standards. However, as demonstrated above, the solicitation of student opinion in the 
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application of Specific Review Standards 1.1 and 1.2 was most helpful. In the absence of the 
numerical rankings and student comments, the author would have chosen different options for 
those Standards. Obtaining student input also invests students in the decision-making process 
and provides the instructor with an opportunity to explain why the course is set up as it is. 
Moreover, it seems more efficient to gather student input directly on course design issues prior to 
the course being offered rather than to try to sift through post-course evaluation comments to 
find comments pertinent to course design. 

CONCLUSIONS AND FUTURE STUDY 

In agreement with the literature, the author found it very useful to begin to apply the QM 
Rubric to a business course that uses online materials and that the author has taught as a web- 
facilitated course, a hybrid/blended course, and as an online course. The author recommends the 
Rubric to other faculty members who want to improve their courses via self-evaluation, 
institutional internal review, and/or the QM external review and certification process. 

The author further recommends that instructors who apply the QM Rubric seek input 
from their students in doing so. Instructors will not always be able to predict which online 
features students will favor, and students will appreciate the opportunity to contribute to course 
design. 

In the future, the author will continue to seek the input of students in applying the 
remaining standards of the QM Rubric to business courses. Moreover, the author will carefully 
review post-course student evaluations of the online materials and courses reviewed with the 
Rubric. One of the self-admitted limitations of the Rubric is that it deals primarily with course 
design, which is a key (but not exclusive) factor of course success. Other factors of course 
success include course delivery, course content, institutional infrastructure, learning management 
system, faculty readiness, and student readiness. (MarylandOnline, 2013) The author also plans 
to analyze whether student opinions of certain features and changes initiated by QM review are 
the same or different based on the students’ gender, class rank, age, or major. 
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